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After-School Reading 

A reading program, adapted for use in the 
after-school setting, resulted in 48 more 
hours of reading instruction during the 
school year (20 percent more) for reading 
program students than for their coun- 
terparts, but it did not increase student 
achievement in reading. The impacts did 
not vary significantly in different grades 
or for students with different prior perfor- 
mance. Nor did the reading program stu- 
dents and their counterparts differ signifi- 
cantly on other in-school behaviors. 




After-school programs receive federal support through the 
21st Century Community Learning Centers, established in 
1999. A primary purpose of the program is to provide op- 
portunities for academic enrichment to help students meet 
state and local standards in core content areas. Findings 
from a previous national evaluation of the program indicate 
that the program grants awarded between 1999 and 2002 
had, on average, a limited academic impact on the academic 
achievement of participating elementary school students. 

A possible factor is that most academic activities at the 
evaluation sites consisted of homework sessions in which 
students received limited additional academic assistance 
(such as instruction or assistance with homework). In ad- 
dition, attendance was limited and sporadic. But analyses 
comparing the academic outcomes of frequent and infre- 
quent participants suggest that increasing attendance alone 
is unlikely to improve the academic findings. So, the lim- 
ited academic effects and the low levels of formal academic 
assistance offered in these programs highlight the need for 
better academic programming. In response, the Institute of 
Education Sciences supported the development and evalu- 
ation of instructional resources in core content areas that 
could be used in after-school programs. 



This brief presents findings for the first of two years of pro- 
gram operations (school year 2005-06) on a study for reading. 

The reading program 

The curriculum developer — Success for All Foundation — 
was selected through a competitive process to adapt its 
school-day reading materials for use in the after-school 
setting. The developer was asked to create material that is 
engaging for students, tied to academic standards, ap- 
propriate for students from diverse economic and social 
backgrounds, and fairly easy for teachers to use with a 
small amount of preparation time. 

The Success for All Foundation adapted its existing school- 
day reading program to create Adventure Island, a struc- 
tured reading model with daily, 45-minute lessons that 
involve switching quickly from one teacher-led activity to 
the next. It includes the key components of effective read- 
ing instruction identified by the National Reading Panel 
and builds cooperative learning into its daily classroom 
routines, which also include reading a variety of selected 
books and frequent assessments built into lessons to moni- 
tor individual student progress. 




2 



AFTER-SCHOOL READING 



The study 

The reading program was implemented in 25 after-school 
centers, which were chosen based on their expressed 
interest and their ability to implement the program and 
research design. The study sample of 1,828 students, in 
grades 2 through 5, were identified by local staff as in need 
of supplemental academic support to meet local academic 
standards and were enrolled in the after-school programs. 
Students were assigned by lottery to either the Success for 
All Foundation’s Adventure Island reading program or to 
the regular after-school program. The regular program con- 
sisted primarily of help with homework or locally assembled 
materials that do not follow a structured curriculum. 

The evaluation examines four primary questions: 

• Does the Adventure Island after-school reading pro- 
gram improve reading proficiency over what students 
would achieve in regular after-school programs, as 
measured by test scores? 

• What are the impacts of the after-school reading 
instruction for subgroups of students based on their 
prior academic performance and grade? 

• Does the after-school reading instruction affect other 
in-school academic behavior outcomes, as measured 
by reports from regular-school-day teachers of student 
engagement, behavior, and homework completion? 

• What does program implementation look like, and was 
it implemented as intended? 

The second and fourth questions address information to 
better target and implement the intervention. The third 
question addresses whether extended learning leads to ad- 
ditional positive or negative student academic behaviors. 

Impact findings from the first year are based on data col- 
lected from students, regular-school-day teachers, and 
school records. The Stanford Achievement Test, Tenth Edi- 
tion (SAT 10), abbreviated battery for reading, was admin- 
istered to students at the beginning and end of the school 
year to measure the gains in achievement. For students 
in grades 2 and 3, the Dynamic Indicators of Basic Early 
Literacy Skills (DIBELS) was also administered to measure 



fluency. A survey of regular-school-day teachers was used 
to measure student academic behavior. The study also 
collected information about program implementation and 
student attendance. 

The study's findings after one year 

In the first year of the study, Adventure Island, the reading 
model put in place in 25 after-school centers, produced the 
following interim findings. 

Student reading skills and other academic behaviors 

Students in the reading program did not experience a 
statistically significant impact on their performance on the 
SAT 10 reading test, overall or on any of the three sub- 
components. There were positive and statistically signifi- 
cant program impacts on one of the two measures in the 
DIBELS fluency test, but after accounting for multiple com- 
parisons, this finding is no longer statistically significant. 
The program had no positive or negative effects on teacher 
reported student behaviors during the school day. 

• The average total SAT 10 reading score of the Ad- 
venture Island group increased over the school year 
by 22.6 scaled points, compared with an increase of 
23.2 scaled points for the other group, resulting in a 
difference that is not statistically significant. There 
were also no statistically significant impacts on the 
three subcomponents of the SAT 10 — vocabulary, read- 
ing comprehension, and word study skills (figure 1). 

• The estimated difference between the Adventure Island 
group and the other group was a statistically signifi- 
cant 3.7 points (effect size = 0.12) in the nonsense 
word fluency subtest of DIBELS. But after accounting 
for multiple comparisons, this finding is no longer 
statistically significant. For the oral fluency subtest of 
DIBELS, the difference of 2.3 points (effect size = 0.07) 
was not statistically significant. 

• Reading score impacts were not statistically significant 
within or between any of the subgroups — by prior 
achievement or by grade. 

• The reading program did not produce statistically sig- 
nificant impacts on any of the three academic behavior 
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FIGURE 1 

Student growth on reading test scores 

Growth from baseline (scaled score points) 




Total Vocabulary Reading Word study 

comprehension skills 

measures: homework completion, attentiveness in 
class, or classroom disruptiveness. 

Program implementation 

The strategies supporting the reading intervention were im- 
plemented as intended, but teacher feedback and program 
observations indicated some teacher implementation chal- 
lenges. Student program participation led to an estimated 
increase in reading instruction of 20 percent more hours of 
reading instruction (48 hours) during the school year. 

• Of the reading program staff, 99 percent were certified 
teachers, most often with three or more years of teach- 
ing experience. More than 80 percent reported daily 
preparation of at least 30 minutes. The average student- 
to-staff ratio of 9:1 was as intended. 

• All 50 after-school teachers interviewed reported ex- 
periencing some challenges related to pacing. Sev- 
enty-four percent of observed program observations 



indicated that teachers did not complete all the compo- 
nents of the lessons in the allotted time; 84 percent of 
the observed teachers did not model or practice fluency 
as was intended during the lesson. 

• On average, the reading program students attended the 
after-school program for 70 days (73 percent of the in- 
struction days offered), which was 7 more days than the 
other group. This translated into an increase in reading 
instruction of 48 hours (55 hours for reading program 
students compared with 6.5 hours for other students, 
who received normal after-school programming). 

Upcoming report 

The study was expanded to include a second year of imple- 
mentation and data collection in 12 of the original partici- 
pating reading centers. This sample includes students who 
were part of the study in the first year and students who 
were new to the study in the second year, allowing the new 
wave of data collection to shed light both on the cumulative 
impact of the enhanced after-school program on return- 
ing students and on the impact of a more mature program 
on new students. The results will be presented in the final 
report of the evaluation. 

For the full report, please visit: 

http://ies.ed.gov/pubsearch/pubsinfo. 

asp?pubid=NCEE20084021 

Black, A.R., Doolittle, F., Zhu P., Unterman, R., and Gross- 
man, J. B. (2008). The Evaluation of Enhanced Academic 
Instruction in After-School Programs: Findings After the 
First Year of Implementation (NCEE 2008-4021). Wash- 
ington, DC: National Center for Education Evaluation and 
Regional Assistance, Institute of Education Sciences, U.S. 
Department of Education. 



NCEE developed the Evaluation Briefs to offer short 
synopses of complex technical evaluation reports. This 
brief was not prepared by the study authors. 
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